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ABSTRACT 

Distance Learning for Developmental Disabilities Workers project is intended 
to assess the effectiveness of the course materials, operations process, and 
implementation. Designed to provide 96 hours of workplace literacy training 
to 426 developmental disabilities workers in about 80 state-operated 
workplaces, the project is a partnership between the Center for Advanced 
Study of Education and the Civil Service Employees Association, Inc., with 
the New York State Office of Mental Retardation and Developmental 
Disabilities. Observations and interviews would be conducted to evaluate 
effectiveness of the project and components, including samples of e-mail 
communications conducted with students. Interviews of the project director, 
educational technology specialist, coordinators, and project partners would 
be conducted during the initial stage of the project and in the fall. This 
formative evaluation consists of a multi-site study. Critical factors would 
be identified and evaluated by sample site and across sample sites. 
Comparison data would be collected to evaluate whether the access and use of 
the computer as an additional option made a difference in student attitudes 
and achievement. The two of the seven Developmental Disabilities Service 
Offices without computer access would be the control group. A preliminary 
observation criteria list with sample criteria for the physical and human 
environments has been compiled; interview questions have been developed for 
instructors and participants. (YLB) 



CE 077 492 

Markowsky, Mary E. 

Self -Directed Workplace Literacy Distance Learning for 
Developmental Disabilities Workers: External Formative 
Evaluation Design. 

City Univ. of New York, NY. Center for Advanced Study in 
Education . 

Office of Vocational and Adult Education (ED) , Washington, 
DC. National Workplace Literacy Program. 

1997-12-31 

lip.; For related documents, see CE 077 493-494. 

VI 9 8 A4 02 98-95 

Reports - Evaluative (142) 

MFOl/PCOl Plus Postage. 

Adult Basic Education; ♦Caregivers; *Developmental 
Disabilities; *Distance Education; Evaluation Criteria; 
♦Formative Evaluation; Labor Education; Literacy Education; 
Models; Partnerships in Education; Program Effectiveness; 
Unions; *Workplace Literacy 
♦Developmental Aides 



The formative evaluation of the Self -Directed Literacy 



**************************************************************** 4 - 4 - 4 -********* 4 -*** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******************************************************************************** 




6 A LIO i 



<N 

oo 

CN 

<N 

■sj- 



Q 



W 



Self-Directed Workplace Literacy Distance Learning 
for Developmental Disabilties Workers: 
External Formative Evaluation Design 



Mary E. Markowsky, Ed. D. 



US DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

ecAjcational resources information 
1 CENTER (ERIC) 

m This document has been reproduced as 
V received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERl position or policy. 




Center for Advanced Study in Education 

The Graduate School and University Center 
of the City University of New York 




2 



Self-Directed Workplace Literacy Distance Learning 
for Developmental Disabilties Workers: 
External Formative Evaluation Design 



Mary E. Markowsky, Ed. D. 



ETC International 

24 West Railroad Avenue, Suite 224 
Tenafly, New Jersey 07670 



U.S. Department of Education Grant Award #V198A 40298-95 
Project Period 01/01/95-12/31/97 



FORMATIVE EVALUATION DESIGN 



Self-Directed Workplace Literacy Distance Learning 
for Developmental Disabilities Workers 

External Evaluator: Mary E. Markowsky, Ed.D. 



Introduction 

The primary goal of the OMRDD/REACH/CUNY Distance Learning Project is to 
provide a model workplace literacy program to help direct care workers in state- 
operated developmental disabilities facilities adjust to a changing workplace. The 
project will develop a distance learning curriculum model for literacy training 
incorporating a variety of print and educational technology options. This study 
represents an initial effort in determining the efficacy of distance learning of 
workplace literacy skills. 

Overall goals to be achieved by the program: 

• To implement a workplace literacy partnership between a labor organization 
and an institution of higher education 

• To demonstrate, evaluate and promote the institutionalization of a workplace 
literacy training model that combines print, videotapes, electronic mail, 
computer-assisted-learning, teleconferencing, and self-directed learning 
methods for direct care workers in the field of developmental disabilities. 

The project is designed to provide 96 hours of workplace literacy training to 426 
developmental disabilities workers in approximately 80 different workplaces 
( approximately 77 community residences and 3 developmental centers) in seven 
different regions in New York State. Each participant will receive workplace literacy 
instruction and educational counseling for 96 hours over a six month usually in four 
one hour sessions per week 100% release time. 
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Purpose Of the Formative Evaluation 

The formative evaluation study is intended to serve the following general purposes: 

(1) Provide timely feedback to CASE/CUNY project staff regarding the 
effectiveness of the course materials and to identify necessary revisions. 

(2) Evaluate curriculum objectives to determine relevance to project 
goals. 

(3) Review curriculum objectives, evaluate six assessment instruments, 
and student materials including videotapes to correlation to curriculum 
objectives. 

(4) Review draft of the Individualized Learning Plan used with 
participants in relation to program's objectives. 

(5) Evaluate effectiveness of the operations process including the recruitment 
process and conduct of advisory committee to project's objectives. 

(7) Provide evaluation on the implementation of distance learning 
educational technology including the facilitation by instructors, 
staff development, and educational counseling components. 

(8) Evaluate the implementation of the teleconferencing component 
and its effectiveness in workplace literacy instruction. 

This evaluation is designed to determine the impact of incorporating a variety of 
instructional modes on the development of literacy skills, as well as the progress 
teachers and students have made in using distance learning for instruction. 
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Procedures 

Both observations and interviews will be conducted to evaluate the effectiveness of 
the overall project and components including samples of e-mail communications 
conducted with students. Interviews of the project director, educational technology 
specialist, coordinators, and project partners from OMRDD, CSEA< and GOER 
will be conducted during the initial stage of the project and in the Fall. A total of 
twelve days are allocated for site visits to selected work sites during mid-Summer 
and mid-Fail. 

This formative evaluation consists of a multi-site study. Critical factors will be 
identified and evaluated by sample site and across sample sites. One critical factor 
identified is the use of computers for instructional purposes. Five of the seven 
DDSOs have full access to computers for instruction. This constitutes an additional 
variable for instructional modes not available across all sites. Comparison data will 
be collected to evaluate whether the access and use of the computer as an additional 
option made a difference in student attitudes and achievement. The two DDSOs 
without computer access will become the control group. 

Implementation issues will be evaluated according to the five phases: 

1. Planning 

2. Equipment Selection, Preparation, and Installation 

3. Preliminary Training and Ongoing Support 

4. Collection and Analysis of Qualitative Data 

5. Preparation of transitional materials to assure continuation of the 
project's goals and benefits. 

The instruments will focus both on the strengths and weaknesses of the program. 
They will include what participants liked and disliked, on procedures that ran 
smoothly and on procedures that were difficult or inefficient. 

Where relevant, the evaluation will include a comparison across methods. For 
example, the use of information from the interviews and from the electronic logs will 
be correlated to assess the patterns of use for computer-assisted instruction. It is 
important that the project instructors save and maintain their e-mail files that 
record student/instructor communication. These files will be reviewed by the 
external examiner for the purpose of the formative evaluation. 
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On-site Observations 

The initial site visits in each of the seven DDSOs will be conducted at the beginning 
of August, 1995. The second observation will be conducted at the end of October, 
1995. The primary purpose of the site visits is to observe the instructors and learners 
involvement in the distance learning activities, evaluate the efficiency and 
effectiveness of educational technology operations. 

The proposed schedule and an observation guide will be used primarily to gather 
comparable data across sites. If in each site or with each subject similar data is 
collected some statements can be made concerning the distribution of facts that are 
gathered. 

The following provides a preliminary Observation Criteria List for the first site visit. 
The Observation Criteria List for the second site visit will be compiled based upon 
the outcomes of the initial visit. 

Observational Criteria is based upon obtaining field notes concerning the physical 
environment and human environment factors. Sample criteria includes: 

Physical Environment 



What is the general appearance of the facility? 

How is the space arranged as a whole? 

Are the computers and/or equipment easily accessible? 
Are study areas defined? 

Human Environment 



What are the various titles of the people who work in the facility? 
What are the jobs of the various specialists? 

How can you tell what positions people hold? 
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What are the specific qualifications for various staff positions? 

What training do staff members receive before assuming their responsibilities? 
What reasons do staff members give for working in the school? 

What is the staff definition of their involvement? 

How are students seen by the stafT? 

What are the students' perspectives on training and work? 

What are they students' perceptions as to why they think they are in the program? 
Analyzing Observational Data 

This data will be prepared in a format to guide recommended module revisions and 
preferred implementation strategies. Procedures will be developed for data analysis 
including techniques to code, chart, and interpret observations. 

Interview Questions 

The following questions will be used in conducting the interviews of the instructors 
and participants, 

Instructors 

How frequently were the on-line services used by the students? 

Did the frequent communication with trainers through e-mail make 
a difference in the performance of students? 

How did the limited number of terminals available on the network 
effect the process? 

Was more equipment acquired as a result of this project? 

Have you observed a difference in the incident reports submitted? 
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How have the Direct Care Workers demonstrated a better understanding of the 
regulations? 

Did you find your comments via e-mail different from your normal communications 
with students? 

Is the sequence of skills and knowledge effective for student learning? 

Does the curriculum achieve its objectives? 

Participants 

Did you find the use of choices empowering? 

Which technology or learning materials did you prefer to use? Why? 

How often did you experience any problems in accessing 
e-mail? What types of problems occurred? 

How long did you typically have to wait to make contact with the computer to which 
you were sending an e-mail message? 

Did you ever need to contact your instructor, but was unable to do so? What was 
the reason? How did this affect your attitude? How did this effect the success of 
your assignment? 

What type of feedback from your instructor did you find most 
helpful? 

Did you ever consult with your instructor when writing your Individualized 
Learning Plan? 

Did you find the core instruction you received helpful in designing 
your ILP? 

Did the knowledge of the electronic monitoring of your e-mail influence what you 
wrote? If yes, why and how? 
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How did receiving instruction at your place of employment impact 
your attendance and your learning? 

What types of assistance did you receive via telephone was the most helpful from 
your instructor during the first weeks of training? 

How often did you speak to your CSFA Advisor by phone? For what purpose? 
Instructional Materials 

Did the orientation videotape prepare you effectively for the lesson? 

Was the workplace literacy handbook easy to read? 

Were the instructions clear and easy to follow? 

If you were asked to write the handbook, what would you change or add? Why? 

How were the video simulations similar to your actual work experience? If not, what 
would you change or add? 

How has the training program contributed to the effectiveness of your work? 

What component of the training sessions worked best? 

What aspects did not seem effective? 

How would you change the approach in the future? 

What support strategies has the administration put in place to support 
meaningful use of technology? 

Which learning options did you chose to use the most? Why? 

How often did you need guidance in choosing your learning path? 

Did you have access to a terminal for e-mail when you needed it? 
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For sites with computers 

Did having one computer at your site effect in any way how often you used it for 
writing exercises? 

Did you observe a difference in your writing when working with 
the word processor? If yes, describe the difference in your 
writing ability. 

Did you use software? If yes, what type of software did you find most useful? 

Was the software easy to use or did you need assistance? 

Are your incident reports more complete in description and more 
concise as a result of this training? If yes, what made the difference? 

Can you better understand the regulations related to your job? 

Did your instructor meet your expectations? 

In which ways was your instructor most helpful? 

ON-GOING EVALUATION 



Review and evaluation of curriculum materials, assessment instruments, and 
individualized education plans as per schedule provided by the project guidelines. 
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